Papers with Continual Pre-training
A Novel Paradigm Boosting Translation Capabilities of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on LLMs focused on supervised fine-tuning but their effectiveness has been limited. |
| Approach: | They propose a paradigm consisting of three stages: Secondary Pre-training using extensive monolingual data, Continual Pre- training with interlinear text format documents, and Leveraging source-language consistent instruction for supervised fine-tuning. |
| Outcome: | The proposed approach surpasses previous work and achieves superior performance compared to models such as NLLB-54B(CITATION) and GPT3.5-text-davinci-003. |
CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search (2025.naacl-industry)
Copied to clipboard
Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Xu Jia, Zhongyi Liu, Jinjie Gu, Yuan Zhou, Linjian Mo
| Challenge: | Relevance modeling between queries and items is a key component of commercial search engines. |
| Approach: | They propose a framework for continual pre-training of LLMs to enhance domain knowledge . they employ queries and multi-field item to jointly pre-train for enhancing domain knowledge. |
| Outcome: | The proposed model achieves convincing performance compared to strong baselines. |
How to Train a Real-World Silicon Concierge? Internalizing Complex Business Workflow to Only OneModel (2026.acl-industry)
Copied to clipboard
| Challenge: | Traditional industrial agents rely on modular workflows that fracture into a labyrinth of ad-hoc patches, leading to cascading errors and high latency. |
| Approach: | They propose a paradigm shift from external workflows to internalized knowledge representation that consolidates complex business logic and SOPs directly into the model’s parameters. |
| Outcome: | The proposed model breaks the impossible triangle of latency, accuracy, and complexity. |
Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities across a wide range of domains, but their generalpurpose pre-training objectives often leave them illsuited for specialized applications such as healthcare. |
| Approach: | They propose a perplexity-aware data scaling law that establishes a predictive relationship between the perplexities of domain-specific data and the test loss. |
| Outcome: | Experiments on medical and general-domain benchmarks show that the proposed scaling law consistently identifies near-optimal training subsets with significantly reduced data consumption. |
Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models capture factual knowledge across a wide range of domains, but refining their capabilities on previously seen knowledge remains a challenge. |
| Approach: | They propose a synthetic knowledge ingestion method that leverages fine-grained synthesis and interleaved generation to construct high-quality data representations from raw knowledge sources. |
| Outcome: | The proposed method outperforms baseline methods on question-answering tasks spanning finance, biomedicine, and open-generation domains. |
CSRP: Chain-of-Thought Reasoning for Chinese Text Correction via Reinforcement Learning with Efficiency-Aware Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models lack specialized priors for subtle grammatical distinctions, and Supervised Fine-Tuning fails to optimize for precision-focused metrics. |
| Approach: | They propose a framework that builds correction capability through Continual Pre-training on 5.9M balanced samples to internalize domain knowledge. |
| Outcome: | The proposed framework outperforms existing models on the NACGEC benchmark with 50.99 F0.5 and 57.17 precision while mitigating over-correction bias. |